{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "TPU",
    "colab": {
      "name": "Simple Classification Model using Keras on Colab TPU.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "N6ZDpd9XzFeN"
      ],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "N6ZDpd9XzFeN"
      },
      "source": [
        "##### Copyright 2020 The TensorFlow Hub Authors.\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "colab_type": "code",
        "id": "KUu4vOt5zI9d",
        "colab": {}
      },
      "source": [
        "# Copyright 2018 The TensorFlow Hub Authors. All Rights Reserved.\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     http://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License.\n",
        "# =============================================================================="
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "a0TLC65g9W5n"
      },
      "source": [
        "# A simple classification model using Keras with Cloud TPUs\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "0CmchfI_AZdY"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This notebook shows how to use Keras to build a simple classification model. The model can train, evaluate, and generate predictions using Cloud TPUs. It uses the iris dataset to predict the species of the flower and also shows how to use your own data instead of using pre-loaded data. This model uses 4 input features ***(SepalLength, SepalWidth, PetalLength, PetalWidth)*** to determine one of these flower species ***(Setosa, Versicolor, Virginica)***.\n",
        "\n",
        "The model trains for 50 epochs and completes in approximately 2 minutes.\n",
        "\n",
        "This notebook is hosted on GitHub. To view it in its original repository, after opening the notebook, select **File > View on GitHub**.\n",
        "\n",
        "\n",
        "**NOTE:** This tutorial is designed to show how to write a simple model using Keras. It should not be used for comparision with training on CPU's because of the very small amount of data being used."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "dgAHfQtuhddd"
      },
      "source": [
        "## Learning objectives\n",
        "\n",
        "In this Colab, you will learn how to:\n",
        "*   Define a Keras model with 2 hidden layers and 10 nodes in each layer.\n",
        "*   Create and compile a Keras model on TPU with a distribution strategy.\n",
        "*   Train, evaluate, and and generate predictions on Cloud TPU.\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "6b8OVJWtPID9"
      },
      "source": [
        "## Instructions\n",
        "\n",
        "<h3>  &nbsp;&nbsp;Train on TPU&nbsp;&nbsp; <a href=\"https://cloud.google.com/tpu/\"><img valign=\"middle\" src=\"https://raw.githubusercontent.com/GoogleCloudPlatform/tensorflow-without-a-phd/master/tensorflow-rl-pong/images/tpu-hexagon.png\" width=\"50\"></a></h3>\n",
        "\n",
        "   1. On the main menu, click Runtime and select **Change runtime type**. Set \"TPU\" as the hardware accelerator.\n",
        "   1. Click Runtime again and select **Runtime > Run All**. You can also run the cells manually with Shift-ENTER. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Lvo0t7XVIkWZ"
      },
      "source": [
        "## Data, model, and training"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ACiNzAp8AoWR"
      },
      "source": [
        "### Imports"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "IgQge6h7AqDw",
        "colab": {}
      },
      "source": [
        "import json\n",
        "import os\n",
        "import pandas as pd\n",
        "import pprint\n",
        "import tensorflow as tf\n",
        "import time\n",
        "import numpy as np\n",
        "from tensorflow import keras\n"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "AFZEq6epY-Sy",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "3074ac12-fdc0-42b8-c85d-060ae610fcc1"
      },
      "source": [
        "print(tf.__version__)\n",
        "import distutils\n",
        "if distutils.version.LooseVersion(tf.__version__) < '1.14':\n",
        "    raise Exception('This notebook is compatible with TensorFlow 1.14 or higher, for TensorFlow 1.13 or lower please use the previous version at https://github.com/tensorflow/tpu/blob/r1.13/tools/colab/classification_iris_data_with_keras.ipynb')"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2.2.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "7FCdy1aBAsXe"
      },
      "source": [
        "### Resolve TPU Address"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "phzyD8iCAzcp",
        "colab": {}
      },
      "source": [
        "use_tpu = True #@param {type:\"boolean\"}\n",
        "\n",
        "if use_tpu:\n",
        "    assert 'COLAB_TPU_ADDR' in os.environ, 'Missing TPU; did you request a TPU in Notebook Settings?'\n",
        "\n",
        "if 'COLAB_TPU_ADDR' in os.environ:\n",
        "  TF_MASTER = 'grpc://{}'.format(os.environ['COLAB_TPU_ADDR'])\n",
        "else:\n",
        "  TF_MASTER=''"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "r0AEjD1vKGXO"
      },
      "source": [
        "### FLAGS used as model params"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "0tcdTWw1KLiF",
        "colab": {}
      },
      "source": [
        "# Model specific parameters\n",
        "\n",
        "# TPU address\n",
        "tpu_address = TF_MASTER\n",
        "\n",
        "# Number of epochs\n",
        "epochs = 50\n",
        "\n",
        "# Number of steps_per_epoch\n",
        "steps_per_epoch = 5\n",
        "\n",
        "# NOTE: Total number of training steps = Number of epochs * Number of steps_per_epochs"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "I3Ckb7SEKSGO"
      },
      "source": [
        "### Download training input data and define prediction input & output"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Y1qZ3cWyKb-b",
        "colab": {}
      },
      "source": [
        "TRAIN_URL = \"http://download.tensorflow.org/data/iris_training.csv\"\n",
        "TEST_URL = \"http://download.tensorflow.org/data/iris_test.csv\"\n",
        "\n",
        "CSV_COLUMN_NAMES = ['SepalLength', 'SepalWidth',\n",
        "                    'PetalLength', 'PetalWidth', 'Species']\n",
        "SPECIES = ['Setosa', 'Versicolor', 'Virginica']\n",
        "\n",
        "PREDICTION_INPUT_DATA = {\n",
        "    'SepalLength': [6.9, 5.1, 5.9, 6.0, 5.5, 6.2, 5.5, 6.3],\n",
        "    'SepalWidth': [3.1, 3.3, 3.0, 3.4, 2.5, 2.9, 4.2, 2.8],\n",
        "    'PetalLength': [5.4, 1.7, 4.2, 4.5, 4.0, 4.3, 1.4, 5.1],\n",
        "    'PetalWidth': [2.1, 0.5, 1.5, 1.6, 1.3, 1.3, 0.2, 1.5],\n",
        "}\n",
        "\n",
        "PREDICTION_OUTPUT_DATA = ['Virginica', 'Setosa', 'Versicolor', 'Versicolor', 'Versicolor', 'Versicolor', 'Setosa', 'Virginica']\n",
        "\n",
        "def maybe_download():\n",
        "    train_path = tf.keras.utils.get_file(TRAIN_URL.split('/')[-1], TRAIN_URL)\n",
        "    test_path = tf.keras.utils.get_file(TEST_URL.split('/')[-1], TEST_URL)\n",
        "\n",
        "    return train_path, test_path\n",
        "\n",
        "def load_data(y_name='Species'):\n",
        "    \"\"\"Returns the iris dataset as (train_x, train_y), (test_x, test_y).\"\"\"\n",
        "    train_path, test_path = maybe_download()\n",
        "\n",
        "    train = pd.read_csv(train_path, names=CSV_COLUMN_NAMES, header=0, dtype={'SepalLength': pd.np.float32,\n",
        "        'SepalWidth': pd.np.float32, 'PetalLength': pd.np.float32, 'PetalWidth': pd.np.float32, 'Species': pd.np.int32})\n",
        "    train_x, train_y = train, train.pop(y_name)\n",
        "\n",
        "    test = pd.read_csv(test_path, names=CSV_COLUMN_NAMES, header=0, dtype={'SepalLength': pd.np.float32,\n",
        "        'SepalWidth': pd.np.float32, 'PetalLength': pd.np.float32, 'PetalWidth': pd.np.float32, 'Species': pd.np.int32})\n",
        "    test_x, test_y = test, test.pop(y_name)\n",
        "\n",
        "    return (train_x, train_y), (test_x, test_y)"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "3UhOWs26pEaq"
      },
      "source": [
        "### Define a Keras model (2 hidden layers with 10 neurons in each)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "OyFLGbBcLT8e",
        "colab": {}
      },
      "source": [
        "def get_model():\n",
        "  return keras.Sequential([\n",
        "    keras.layers.Dense(10, input_shape=(4,), activation=tf.nn.relu, name = \"Dense_1\"),\n",
        "    keras.layers.Dense(10, activation=tf.nn.relu, name = \"Dense_2\"),\n",
        "    keras.layers.Dense(3, activation=None, name = \"logits\"),\n",
        "    keras.layers.Dense(3, activation=tf.nn.softmax, name = \"softmax\")\n",
        "  ])"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "yV9uzDL3bBLG"
      },
      "source": [
        "### Compiling the model with a distribution strategy\n",
        "To make the model usable by a TPU, we first must create and compile it using a distribution strategy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ZbwW0pDxFQyx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 708
        },
        "outputId": "95d7ceb6-8063-41b4-a2f7-be4947e5f2fe"
      },
      "source": [
        "resolver = tf.distribute.cluster_resolver.TPUClusterResolver(TF_MASTER)\n",
        "tf.config.experimental_connect_to_cluster(resolver)\n",
        "tf.tpu.experimental.initialize_tpu_system(resolver)\n",
        "strategy = tf.distribute.experimental.TPUStrategy(resolver)\n"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Initializing the TPU system: grpc://10.110.91.34:8470\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Initializing the TPU system: grpc://10.110.91.34:8470\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Clearing out eager caches\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Clearing out eager caches\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished initializing TPU system.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished initializing TPU system.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Found TPU system:\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Found TPU system:\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Cores: 8\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Cores: 8\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Workers: 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Workers: 1\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Cores Per Worker: 8\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Cores Per Worker: 8\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:localhost/replica:0/task:0/device:CPU:0, CPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:localhost/replica:0/task:0/device:CPU:0, CPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:localhost/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:localhost/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 0, 0)\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "2xnX-NTsFQQa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 294
        },
        "outputId": "aa5d6487-34cd-47f6-f5fa-0598b4e0bfaf"
      },
      "source": [
        "with strategy.scope():\n",
        "  model = get_model()\n",
        "  model.compile(optimizer=tf.keras.optimizers.Adagrad(learning_rate=0.1), \n",
        "                loss=tf.keras.losses.sparse_categorical_crossentropy, \n",
        "                metrics=['accuracy'])\n",
        "\n",
        "model.summary()"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "Dense_1 (Dense)              (None, 10)                50        \n",
            "_________________________________________________________________\n",
            "Dense_2 (Dense)              (None, 10)                110       \n",
            "_________________________________________________________________\n",
            "logits (Dense)               (None, 3)                 33        \n",
            "_________________________________________________________________\n",
            "softmax (Dense)              (None, 3)                 12        \n",
            "=================================================================\n",
            "Total params: 205\n",
            "Trainable params: 205\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "cMKySsFRFzlg"
      },
      "source": [
        "### Train the model on TPU"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "9EoghGTOUvyz",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "ee2db248-39e6-4589-cad5-913d4ed8cf6b"
      },
      "source": [
        "# Fetch the data\n",
        "(train_x, train_y), (test_x, test_y) = load_data()\n",
        "\n",
        "# Train the model\n",
        "model.fit(\n",
        "  train_x.values, train_y.values,\n",
        "  steps_per_epoch = steps_per_epoch,\n",
        "  epochs=epochs,\n",
        ")"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from http://download.tensorflow.org/data/iris_training.csv\n",
            "\r8192/2194 [================================================================================================================] - 0s 0us/step\n",
            "Downloading data from http://download.tensorflow.org/data/iris_test.csv\n",
            "8192/573 [============================================================================================================================================================================================================================================================================================================================================================================================================================================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:27: FutureWarning: The pandas.np module is deprecated and will be removed from pandas in a future version. Import numpy directly instead\n",
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:28: FutureWarning: The pandas.np module is deprecated and will be removed from pandas in a future version. Import numpy directly instead\n",
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:31: FutureWarning: The pandas.np module is deprecated and will be removed from pandas in a future version. Import numpy directly instead\n",
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:32: FutureWarning: The pandas.np module is deprecated and will be removed from pandas in a future version. Import numpy directly instead\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/50\n",
            "5/5 [==============================] - 0s 7ms/step - accuracy: 0.4667 - loss: 1.1191\n",
            "Epoch 2/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.7250 - loss: 0.7990\n",
            "Epoch 3/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.7000 - loss: 0.5524\n",
            "Epoch 4/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.7083 - loss: 0.4898\n",
            "Epoch 5/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.7083 - loss: 0.4448\n",
            "Epoch 6/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.7417 - loss: 0.4312\n",
            "Epoch 7/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.7333 - loss: 0.4175\n",
            "Epoch 8/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9083 - loss: 0.4045\n",
            "Epoch 9/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.7833 - loss: 0.3834\n",
            "Epoch 10/50\n",
            "5/5 [==============================] - 0s 7ms/step - accuracy: 0.8750 - loss: 0.3639\n",
            "Epoch 11/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.8250 - loss: 0.3659\n",
            "Epoch 12/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.8750 - loss: 0.3493\n",
            "Epoch 13/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.8917 - loss: 0.3038\n",
            "Epoch 14/50\n",
            "5/5 [==============================] - 0s 7ms/step - accuracy: 0.9250 - loss: 0.2926\n",
            "Epoch 15/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9333 - loss: 0.2582\n",
            "Epoch 16/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9250 - loss: 0.2462\n",
            "Epoch 17/50\n",
            "5/5 [==============================] - 0s 7ms/step - accuracy: 0.9417 - loss: 0.2132\n",
            "Epoch 18/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9417 - loss: 0.2183\n",
            "Epoch 19/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9000 - loss: 0.2533\n",
            "Epoch 20/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9417 - loss: 0.1799\n",
            "Epoch 21/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9000 - loss: 0.2484\n",
            "Epoch 22/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9167 - loss: 0.2163\n",
            "Epoch 23/50\n",
            "5/5 [==============================] - 0s 7ms/step - accuracy: 0.9583 - loss: 0.1751\n",
            "Epoch 24/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9250 - loss: 0.1933\n",
            "Epoch 25/50\n",
            "5/5 [==============================] - 0s 7ms/step - accuracy: 0.9667 - loss: 0.1479\n",
            "Epoch 26/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9500 - loss: 0.1362\n",
            "Epoch 27/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9833 - loss: 0.1257\n",
            "Epoch 28/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9667 - loss: 0.1170\n",
            "Epoch 29/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9667 - loss: 0.1428\n",
            "Epoch 30/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9750 - loss: 0.1060\n",
            "Epoch 31/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9750 - loss: 0.1274\n",
            "Epoch 32/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9667 - loss: 0.1236\n",
            "Epoch 33/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9583 - loss: 0.0952\n",
            "Epoch 34/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9750 - loss: 0.0998\n",
            "Epoch 35/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9750 - loss: 0.0883\n",
            "Epoch 36/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9667 - loss: 0.1000\n",
            "Epoch 37/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9750 - loss: 0.0942\n",
            "Epoch 38/50\n",
            "5/5 [==============================] - 0s 7ms/step - accuracy: 0.9500 - loss: 0.1247\n",
            "Epoch 39/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9833 - loss: 0.0843\n",
            "Epoch 40/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9750 - loss: 0.0798\n",
            "Epoch 41/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9667 - loss: 0.0967\n",
            "Epoch 42/50\n",
            "5/5 [==============================] - 0s 7ms/step - accuracy: 0.9667 - loss: 0.1272\n",
            "Epoch 43/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9500 - loss: 0.1022\n",
            "Epoch 44/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9667 - loss: 0.0886\n",
            "Epoch 45/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9833 - loss: 0.0767\n",
            "Epoch 46/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9833 - loss: 0.0804\n",
            "Epoch 47/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9833 - loss: 0.0850\n",
            "Epoch 48/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9750 - loss: 0.0906\n",
            "Epoch 49/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9833 - loss: 0.0989\n",
            "Epoch 50/50\n",
            "5/5 [==============================] - 0s 6ms/step - accuracy: 0.9667 - loss: 0.0911\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f1c7383cc88>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "zC-VT-cGpl0A"
      },
      "source": [
        "### Evaluation of the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "xto_EAjm1GdC",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "e65be084-d79e-46fb-88c6-8aeb0e65e595"
      },
      "source": [
        "model.evaluate(test_x.values, test_y.values,\n",
        "    batch_size=8)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "4/4 [==============================] - 0s 10ms/step - accuracy: 0.9333 - loss: 0.0730\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.07300268858671188, 0.9333333969116211]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "VDX-D0fSpqbf"
      },
      "source": [
        "### Save the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Cvpg8f-E2FsK",
        "colab": {}
      },
      "source": [
        "model.save_weights('./DNN_TPU_1024.h5', overwrite=True)"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "1vAGtvL9qz2t"
      },
      "source": [
        "## Prediction"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "2cM1O_2cq4pU"
      },
      "source": [
        "### Prediction data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "i6qpnrapbiEy",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 173
        },
        "outputId": "371f967f-7b63-47c4-9fcf-016996f2bb11"
      },
      "source": [
        "COLUMNS_NAME=['SepalLength', 'SepalWidth', 'PetalLength', 'PetalWidth']\n",
        "data = pd.DataFrame(PREDICTION_INPUT_DATA, columns=COLUMNS_NAME)\n",
        "print(data)"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "   SepalLength  SepalWidth  PetalLength  PetalWidth\n",
            "0          6.9         3.1          5.4         2.1\n",
            "1          5.1         3.3          1.7         0.5\n",
            "2          5.9         3.0          4.2         1.5\n",
            "3          6.0         3.4          4.5         1.6\n",
            "4          5.5         2.5          4.0         1.3\n",
            "5          6.2         2.9          4.3         1.3\n",
            "6          5.5         4.2          1.4         0.2\n",
            "7          6.3         2.8          5.1         1.5\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "he7_-JNtq82W"
      },
      "source": [
        "### Prediction on TPU"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ZhnrwcSe3KER",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 294
        },
        "outputId": "7e0287b3-7164-4493-fd03-82b09c863c05"
      },
      "source": [
        "predictions = model.predict(data.values.astype(np.float32))\n",
        "template = ('\\nPrediction is \"{}\" ({:.1f}%), expected \"{}\"')\n",
        "for pred_dict, expec in zip(predictions, PREDICTION_OUTPUT_DATA):\n",
        "  class_index = np.argmax(pred_dict)\n",
        "  class_probability = np.max(pred_dict)\n",
        "  print(template.format(SPECIES[class_index], 100*class_probability, expec))"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Prediction is \"Virginica\" (87.8%), expected \"Virginica\"\n",
            "\n",
            "Prediction is \"Setosa\" (99.8%), expected \"Setosa\"\n",
            "\n",
            "Prediction is \"Versicolor\" (97.3%), expected \"Versicolor\"\n",
            "\n",
            "Prediction is \"Versicolor\" (97.3%), expected \"Versicolor\"\n",
            "\n",
            "Prediction is \"Versicolor\" (97.2%), expected \"Versicolor\"\n",
            "\n",
            "Prediction is \"Versicolor\" (97.3%), expected \"Versicolor\"\n",
            "\n",
            "Prediction is \"Setosa\" (100.0%), expected \"Setosa\"\n",
            "\n",
            "Prediction is \"Versicolor\" (55.7%), expected \"Virginica\"\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "DVpuJleArGKH"
      },
      "source": [
        "### Prediction on CPU"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "0DrKs2PGrIhY",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 294
        },
        "outputId": "879e8029-4e5d-42a0-92f5-fd4e2098be3a"
      },
      "source": [
        "cpu_model = get_model()\n",
        "cpu_model.load_weights('./DNN_TPU_1024.h5')\n",
        "cpu_predictions = cpu_model.predict(data)\n",
        "template = ('\\nPrediction is \"{}\" ({:.1f}%), expected \"{}\"')\n",
        "for pred_dict, expec in zip(cpu_predictions, PREDICTION_OUTPUT_DATA):\n",
        "  class_index = np.argmax(pred_dict)\n",
        "  class_probability = np.max(pred_dict)\n",
        "  print(template.format(SPECIES[class_index], 100*class_probability, expec))"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Prediction is \"Virginica\" (87.2%), expected \"Virginica\"\n",
            "\n",
            "Prediction is \"Setosa\" (99.8%), expected \"Setosa\"\n",
            "\n",
            "Prediction is \"Versicolor\" (97.2%), expected \"Versicolor\"\n",
            "\n",
            "Prediction is \"Versicolor\" (97.2%), expected \"Versicolor\"\n",
            "\n",
            "Prediction is \"Versicolor\" (97.2%), expected \"Versicolor\"\n",
            "\n",
            "Prediction is \"Versicolor\" (97.2%), expected \"Versicolor\"\n",
            "\n",
            "Prediction is \"Setosa\" (100.0%), expected \"Setosa\"\n",
            "\n",
            "Prediction is \"Versicolor\" (54.7%), expected \"Virginica\"\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "2a5cGsSTEBQD"
      },
      "source": [
        "## What's next\n",
        "\n",
        "* Learn about [Cloud TPUs](https://cloud.google.com/tpu/docs) that Google designed and optimized specifically to speed up and scale up ML workloads for training and inference and to enable ML engineers and researchers to iterate more quickly.\n",
        "* Explore the range of [Cloud TPU tutorials and Colabs](https://cloud.google.com/tpu/docs/tutorials) to find other examples that can be used when implementing your ML project.\n",
        "\n",
        "On Google Cloud Platform, in addition to GPUs and TPUs available on pre-configured [deep learning VMs](https://cloud.google.com/deep-learning-vm/),  you will find [AutoML](https://cloud.google.com/automl/)*(beta)* for training custom models without writing code and [Cloud ML Engine](https://cloud.google.com/ml-engine/docs/) which will allows you to run parallel trainings and hyperparameter tuning of your custom models on powerful distributed hardware.\n"
      ]
    }
  ]
}